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DETAILED ACTION 
Information Disclosure Statement 

1 . The information disclosure statement filed 3/16/2004 fails to comply with 37 CFR 
1.98(a)(2), which requires a legible copy of each cited foreign patent document; each 
non-patent literature publication or that portion which caused it to be listed; and all other 
information or that portion which caused it to be listed. It has been placed in the 
application file, but the information referred to therein has not been considered. 

The reference that has not been considered is the URL for the Open Source 
Computer Vision Library, http://www.intel.com/research/mrl/research/opencv/ . This 
reference is problematic as it is non-static and the information changes over time. In fact 
the address provided is no longer current. To be considered, a printed copy of the web- 
pages of interest would need to have been submitted in the IDS at the time of 
application. 
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Claim Rejections - 35 USC § 101 

35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

The USPTO "Interim Guidelines for Examination of Patent Applications for Patent 
Subject Matter Eligibility" ( Official Gazette notice of 22 November 2005 ), Annex IV, 
reads as follows: 

Descriptive material can be characterized as either "functional descriptive material" or "nonfunctional 
descriptive material." In this context, "functional descriptive material" consists of data structures and 
computer programs which impart functionality when employed as a computer component. (The 
definition of "data structure" is "a physical or logical relationship among data elements, designed to 
support specific data manipulation functions." The New IEEE Standard Dictionary of Electrical and 
Electronics Terms 308 (5th ed. 1993).) "Nonfunctional descriptive material" includes but is not limited 
to music, literary works and a compilation or mere arrangement of data. 

When functional descriptive material is recorded on some computer-readable medium it becomes 
structurally and functionally interrelated to the medium and will be statutory in most cases since use of 
technology permits the function of the descriptive material to be realized. Compare In re Lowry, 32 
F.3d 1579, 1583-84, 32 USPQ2d 1031, 1035 (Fed. Cir. 1994) (claim to data structure stored on a 
computer readable medium that increases computer efficiency held statutory) and Warmerdam, 33 
F.3d at 1360-61 , 31 USPQ2d at 1759 (claim to computer having a specific data structure stored in 
memory held statutory product-by-process claim) with Warmerdam, 33 F.3d at 1 361 , 31 USPQ2d at 
1760 (claim to a data structure per se held nonstatutory). 

In contrast, a claimed computer-readable medium encoded with a computer program is a computer 
element which defines structural and functional interrelationships between the computer program and 
the rest of the computer which permit the computer program's functionality to be realized, and is thus 
statutory. See Lowry, 32 F.3d at 1583-84, 32 USPQ2d at 1035. 



2. Claims 36-41 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter as follows. Claims 36-41 define "An article 
comprising a storage medium ..." embodying functional descriptive material. However, 
the claim does not define a computer-readable medium or computer-readable memory 
and is thus non-statutory for that reason (i.e., "When functional descriptive material is 
recorded on some computer-readable medium it becomes structurally and functionally 
interrelated to the medium and will be statutory in most cases since use of technology 
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permits the function of the descriptive material to be realized" - Guidelines Annex IV). 
The scope of the presently claimed invention encompasses products that are not 
necessarily computer readable, and thus NOT able to impart any functionality of the 
recited program. The examiner suggests amending the claim(s) to embody the program 
on "computer-readable medium" or equivalent; assuming the specification does NOT 
define the computer readable medium as a "signal", "carrier wave", or "transmission 
medium" which are deemed non-statutory (refer to "note" below). Any amendment to 
the claim should be commensurate with its corresponding disclosure. 

Note: 

A "signal" (or equivalent) embodying functional descriptive material is neither a process 
nor a product (i.e., a tangible "thing") and therefore does not fall within one of the four statutory 
classes of § 101 . Rather, "signal" is a form of energy, in the absence of any physical structure or 
tangible material. 

Should the full scope of the claim as properly read in light of the disclosure 
encompass non-statutory subject matter such as a "signal", the claim as a whole would 
be non-statutory. In the case where the specification defines the computer readable 
medium or memory as statutory tangible products such as a hard drive, ROM, RAM, 
etc, as well as a non-statutory entity such as a "signal", "carrier wave", or "transmission 
medium", the examiner suggests amending the claim to include the disclosed tangible 
computer readable media, while at the same time excluding the intangible media such 
as signals, carrier waves, etc. 
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Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351 (a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

4. Claims 1,3-5, 36-38 are rejected under 35 U.S.C. 102(e) as being anticipated 
by Yuille et al. (cited in applicants IDS). 

Regarding claim 1 , Yuille discloses estimating features of a mouth in a current 
image of a sequence of digital images of a human face ("We define a mouth-closed 
template and a mouth-open template ...", Yuille, page 104, column 1, line 5 and figure 5 
on page 104) by deriving a deformable mouth model template ("We propose a new 
method to detect such features by using deformable templates", Yuille, page 99, column 
1 , line 17) in an iterative process ("The parameters of the template are then updated by 
steepest descent", Yuille, page 99, column 2, line 7, and figure 8 on page 107), said 
process including: 

minimizing an energy function ("The minimum of the energy function corresponds 
to the best fit with the image", Yuille, page 99, column 2, line 6) receiving iteration- 
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dependent arguments to determine optimal transformation parameters of an iteration- 
dependent transformation ("The final values of these parameters can be used to 
describe the features", Yuille, page 99, column 1, line 21); and 

transforming components of said deformable mouth model template by said 
iteration-dependent transformation having said optimal transformation parameters ("The 
final values of these parameters can be used to describe the features." Yuille, page 99, 
column 1, line 21). The parameters referenced by Yuille describe the modifications 
needed modify the template from its base state to a best fit with the image. 

Regarding claim 3, Yuille teaches the elements of claim 1 above wherein 
estimating said features includes estimating positions of lips of said mouth ("The 
parabolas for the top of the upper lip (Pi and P 2 ), bottom of the upper lip (P 3 ), top of the 
lower lip (P 4 ), and bottom of the lower lip (P 5 ) are given by equations (21), (22), (23), 
(24), and (25), respectively", Yuille, page 105, column 2, line 3). The tops and bottoms 
mentioned describe the position of the lips. 

Regarding claim 4, Yuille teaches the elements of claim 1 above wherein 
estimating said features includes estimating shapes of lips of said mouth ("The 
parabolas for the top of the upper lip (P and P 2 ), bottom of the upper lip (P 3 ), top of the 
lower lip (P 4 ), and bottom of the lower lip (P 5 ) are given by equations (21), (22), (23), 
(24), and (25), respectively", Yuille, page 105, column 2, line 3). The parabolas 
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mentioned describe the shape of the lips, the parabola parameters being developed in 
the deformation of the template. 

Regarding claim 5, Yuille teaches the elements of claim 1 above wherein 
estimating said features includes estimating positions of teeth in said mouth ("The 
mouth-open template can be obtained by having two valley parabolas instead of one. 
The region between them corresponds to the teeth", Yuille, page 105, column 2, line 1) 

Regarding claim 36, as described above for claim 1 , Yuille teaches the elements 
of the method as described in claim! Yuille describes the results of his tests as run on 
a computer ("This theory was tested on real images using a SUN4 computer" Yuille, 
page 102, column 1). The instructions and data would of necessity have been on a 
computer readable storage medium. 

Regarding claim 37, Yuille teaches the elements of the method in common with 
claim 1 ,3 and 4. Testing his algorithm would require a computer readable storage 
medium as discussed above for claim 36. 

Regarding claim 38, Yuille teaches the elements of the method in common with 
claim 1 and 5. Testing his algorithm would require a computer readable storage medium 
as discussed above for claim 36. 
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Claim Rejections - 35 USC § 103 



5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claim 2 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yuille et 
al. (cited in applicants IDS) 

Regarding claim 2, Yuille teaches said process further includes: 

initializing said deformable model template according to a base image of 

said sequence ("for succeeding frames, we use as the initial position the best fit 

of the preceding frame", Yuille, page 109. column 1 , line 2). 

Yuille example is for eye tracking rather than mouth tracking in this description, 
but Yuille does state "It is straightforward to adapt the deformable templates for 
tracking", page 1 06, column 2, first sentence under Tracking heading). 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to apply the tracking method of Yuille to the mouth in facial images in 
order to track mouths automatically ("Here we describe a straightforward 
implementation that tracks eyes automatically", Yuille, page 106, column 2, 2 nd 
sentence under Tracking heading). 
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7. Claims 18-20, 24 and 25 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Yuille et al. (cited in applicants IDS) in combination with Yan et al. 
(US 6,975,750). 

Regarding claim 18, Yuille teaches the elements that are in common with claim 1 
as described above. Yuille does not teach [an] energy function [that] includes a bound 
energy term to ensure that said deformable mouth model template represents a 
physiologically possible state of said mouth. 

Yan working in the same field of endeavor of modeling facial features, does 
teach [an] energy function [that] includes a bound energy term to ensure that said 
deformable mouth model template represents a physiologically possible state of said 
mouth ("The creation of the cost function is an important part in the deformable template 
procedure. To this end, different energy items are defined to express the fitting degree 
between the template and the image properties such as the peaks, valleys and edges.", 
Yan, column 7, line 28, and "an infernal constraint function and a punishment function 
are defined", Yan, column 7, line 31). 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to apply the internal constraint and punishment functions of Yan to the 
feature modeling system of Yuille in order "to avoid the template deforming to an illegal 
or unreasonable shape" (Yan, column 7, line 30). 

Regarding claim 19, the combination of Yuille and Yan teaches [t]he method of 
claim 18, wherein said deformable mouth model template includes contours 
representing upper and lower boundaries of an upper lip of said mouth (figure 6, Yuille, 
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page 105, column 2), and contours representing upper and lower boundaries of a lower 

lip of said mouth (figure 6, Yuille, page 105, column 2), 

wherein said energy function is a lips energy objective function, and 

wherein said bound energy term penalizes lip shapes and lip positions that vary 

too greatly from lip shapes and lip positions of said mouth in a base image of said 

sequence ("an internal constraint function and a punishment function are defined", Yan, 

column 7, line 31 ", Yan, column 7 and 8 ). 

Regarding claim 20, the combination of Yuille and Yan teaches The method of 
claim 18, wherein said deformable mouth model template includes inner and outer lip 
contours representing upper and lower lips of said mouth (see above) and teeth 
contours representing upper and lower teeth of said mouth ("the teeth potential, which 
attempts to maximize the average intensity and strength of edges in the teeth region R1 
between the upper and lower lips", Yuille, page 106, column 1 and "The first four 
internal potentials are the same as for the mouth-open template", Yuille, page 106, 
column 1, Yuille is stating that the teeth are modeled much the same as the lips with 
corresponding upper and lower edges), 

wherein said energy function is a teeth energy objective function, and 
wherein said bound energy term penalizes any occurrence of said teeth contours 
overlapping and any occurrence of said teeth contours not located between said inner 
lip contours ("an internal constraint function and a punishment function are defined", 
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Yan, column 7, line 31 , it would be obvious for the reasons given above to apply the 
constraint and punishment to the teeth equations just as for the lip equations. 

Regarding claim 24, the combination of Yuille and Yan teach the elements that are 
common with claim 1 . The combination further teaches wherein said energy function 
includes a teeth gap energy term to describe vertical gaps between the upper teeth and 
lower teeth. (Yuille describes a "valley" term E v to describe the gap between the lips. 
"The mouth-open template can be obtained by having two valley parabolas instead of 
one", Yuille, page 105, column 2, line 1, It would be reasonable to have such a valley 
term to describe any separation between the teeth). 

Regarding claim 25, the combination of Yuille and Yan teaches an energy 
function [that] is a lips energy objective function ("the edge potential, calculated along 

the upper lip parabolae y(x) and y 2 (x) and the lower lip parabola y 3 (x)", Yuille, 
page 105, column 1 ,line 2, the function clearly has lip energy). 

8. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yuille et 
al. (cited in applicants IDS) in combination with Malciu et al. (cited in applicants IDS). 

Regarding claim 21 Yuille teaches all the elements that are common with claim 1. 
Yuille also teaches an energy function [that] includes an energy term to attract contours 
of said deformable mouth model template to respective parabolas. Yuille does not teach 
using an elastic spline term.. 
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Malciu working in the same field of endeavor of facial feature tracking, does 
teach an energy function [that] includes an elastic spline energy term to attract contours 
of said deformable mouth model template to respective parabolas ("spline-based 
representations have been successfully used for facial component modeling" Malciu, 
page 54, line 6, a mouth is a facial component) 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to combine the spline modeling technique of Malciu to the facial feature 
modeling system of Yuille in order to select a mouth template function that is accurate 
and compact ("The selection of an adequate family of shape functions is dictated by 
accuracy and compactness requirements. The most frequent choice consists of second 
order interpolating polynomials, especially parabolic arcs", Malciu, page 54, line 1) 

9. Claim 22 is rejected under 35 U.S.C. 103(a) as being unpatentable over Yuille 
(cited in applicants IDS) and Malciu (cited in applicants IDS) in combination with 
Schoepflin (US 6,574,353 B1). 

Regarding claim 22, Malciu teaches the use of splines for modeling the mouth. 

The combination of Yuille and Malciu does not teach an elastic spline energy 
term [which] is zero for coarse transformations and for fine transformations. 

Schoepflin, working in the same field of endeavor of object tracking via 
deformable templates, does teach an elastic spline energy term [which] is zero for 
coarse transformations ("In a preferred embodiment, three levels of deformation and 
tracking are implemented. At a highest level, an initial template used for a current image 
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frame is translated and rotated to coarsely locate the object boundary among the 
energy representation of the image frame", Schoepflin, column 2, line 1 1 ) and for fine 
transformations ("at a middle level, an affine transformation is implemented to deform 
the template", Schoepflin, column 2, line 27). 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to combine the deformable template hierarchy technique of Schoepflin to 
the facial feature modeling system of the combination of Yuille and Malciu in order to 
accurately track a complex boundary ("an accurate boundary of an object is tracked for 
objects which deform or include rapidly moving sub-portions." Schoepflin, column 13, 
line 41). 

10. Claims 33 and 39 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Yuille et al. (cited in applicants IDS) in combination with Moulton (US 6,778,252 
B2) 

Regarding claim 33 Yuille teaches all the elements that are common with claim 1. 
Yuille does not teach a deformable mouth model template including a first contour to 
represent a lower boundary of upper teeth of said mouth and a second contour to 
represent an upper boundary of lower teeth of said mouth. 

Moulton, working in the same field of endeavor of modeling facial motion, does 
teach a deformable mouth model template including a first contour to represent a lower 
boundary of upper teeth of said mouth ("a point for the bottom edge of the upper teeth", 
Moulton, column 8, line 12) and a second contour to represent an upper boundary of 
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lower teeth ("a point for the upper edge of the lower teeth", Moulton, column 8, line 1 3) 
of said mouth. Moulton indicates that these points are associated with edges. These 
edges correspond to contours ("All these points are connected to surfaces that can be 
recognized as edges by the computer vision system" Moulton, column 8, line 14). 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to combine the teeth contour tracking of Moulton to the facial feature 
modeling system of Yuille in order to generate a more complete model of a subject 
mouth. In the course of normal conversation the teeth are often visible and it would be 
desirable to include in the model. 

Regarding claim 39, the Yuille Moulton combination teaches the elements of the 
method in common with claim 33. . Yuille describes the results of his tests as run on a 
computer ("This theory was tested on real images using a SUN4 computer" Yuille, page 
102, column 1). The instructions and data would of necessity have been on a computer 
readable storage medium. 



11. Claims 34, 35, 40 and 41 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Yuille et al. (cited in applicants IDS) and Moulton (US 6,778,252 B2) 
in combination with Malciu et al. (cited in applicants IDS). 



Application/Control Number: 10/800,732 Page 15 

Art Unit: 2624 

Yuille and Moulton teach the elements of the method of claim 33 as described 
above. Yuille and Moulton do not teach a deformable mouth model template [which] 
further includes a first center control point located substantially on the center of said first 
contour, a second center control point located substantially on the center of said second 
contour, a right control point common to said first contour and said second contour to 
represent a right comer of said mouth, and a left control point common to said first 
contour and said second contour to represent a left corner of said mouth. 

Malciu, does teach a deformable mouth model template [which] further includes a 
first center control point located substantially on the center of said first contour, a 
second center control point located substantially on the center of said second contour, a 
right control point common to said first contour and said second contour to represent a 
right comer of said mouth, and a left control point common to said first contour and said 
second contour to represent a left corner of said mouth (Malciu, figure 1 and 2). 

It would have been obvious at the time the invention was made to one of ordinary 
skill in the art to combine the control points of Malciu with the facial feature modeling 
system of the combination of Yuille and Moulton in order to establish "templates to 
model in a compact fashion highly variable objects with multiple parts and complex 
topologies" (Malciu, page 52, line 23). It also would be desirable to have models 
consistent with the MPEG-4 international standard (Malcui, page 53, first paragraph). 

Regarding claim 35, the combination of Yuille, Mouton and Malcui teaches a 
deformable mouth model template [that] further includes a first left-center control point 
located on said first contour between said left control point and said first center control 
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point, a second left-center control point located on said second contour between said 
left control point and said second center control point, a first right-center control point 
located on said first contour between said right control point and said first center control 
point, and a second left-center control point located on said second contour between 
said right control point and said second center control point (Malciu, figures 1 and 2). 

Regarding claim 40, the combination of Yuille, Mouton and Malciu teaches the 
elements of the method in common with claim 34. Testing his algorithm would require a 
computer readable storage medium as discussed above for claim 36. 

Regarding claim 41 , the combination of Yuille, Mouton and Malciu teaches the 
elements of the method in common with claim 35. Testing his algorithm would require a 
computer readable storage medium as discussed above for claim 36. 

Allowable Subject Matter 

12. Claims 6-17, 23, and 26 are objected to as being dependent upon a rejected 
base claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 

1 3. The following is a statement of reasons for the indication of allowable subject 
matter in the claim listed above: 

Regarding claim 6 as well as claims 7-10 which depend therefrom, the prior art of 
record does not teach double-blurring particular digital images of said sequence to 
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produce double-blurred images; or double-filtering maps derived from said current 
image to produce double-filtered maps, wherein for a particular iteration, minimizing 
said energy function includes minimizing said energy function receiving said double- 
blurred images and said double-filtered maps. 

Regarding claim 1 1 as well as claims 12-16 which depend therefrom, the prior art 
of record does not teach blurring particular digital images of said sequence to produce 
blurred images; and filtering maps derived from said current image to produce.filtered 
maps, wherein for a particular iteration, minimizing said energy function includes 
minimizing said energy function receiving said blurred images and said filtered maps, 
and said iteration-dependent transformation is a fine transformation. 

Regarding claim 23, the prior art of record does not teach a lips energy objective 
function and said elastic spline energy term is related to a square of a width of said 
mouth in a base image of said sequence. 

Regarding claim 26, the prior art of record does not teach [a] teeth gap energy 
term [that] also describes vertical edges of teeth and an absence of teeth in a cavity of 
said mouth. 
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1 4. Claims 27-32 are allowed. 
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The following is an examiner's statement of reasons for allowance for the claims 
listed above: 

Regarding claim 27 as well as claims 28 and 29, which depend therefrom, the 
prior art of record does teach the use of texture energy (Malciu, page 55, 3 rd 
paragraph), it does not teach the use of texture differences in lips and corners of said 
mouth compared to a different image of said sequence. 

Regarding claim 30, as well as claims 31 and 32 that depend therefrom, the prior 
art of record does not teach an energy function [that] includes a corner energy term that 
attracts lip corners to an area having a particular vertical intensity gradient structure. 

Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 

Conclusion 

1 5. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Kervrann et al. (A Hierarchical Markov Modeling Approach for the Segmentation 
of Tracking of Deformable Shapes) does teach using gradients in the tracking of 
deformable shapes (page 174, column 1, 3 rd paragraph). 
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Mirhosseini et al. (A hierarchical and adaptive Deformable Model for Mouth 
Boundary Detection) teaches mouth corner detection, but does not teach texture 
differences or vertical intensity gradients. 

Freeman (US 5,454,043) teaches blurring a histogram signature to ease pattern 
matching, but he does not describe additional filtering. 

Berger (US 2005/0057569 A1) teaches using a base model as a starting point for 
a deformable model. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Thomas M. Redding whose telephone number is (571 ) 
270-1579. The examiner can normally be reached on Mon - Fri 7:30 am - 5:00 pm EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Brian P. Werner can be reached on (571) 272-7401. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/TMR/ 




